Repository navigation
docs(research): Measure which Harness tools survive an ACP attach - #788
Merged
Merged
Conversation
5 tasks done
omesser
force-pushed
the
spike/668-harness-tools-under-acp
branch
2 times, most recently
from
September 16, 2026 21:06
7c46acd to
e93e5d9
Compare
Live probes against claude-agent-acp 0.78.0, grok, hermes, opencode and pi-acp, recorded with Fact / Inference / Assumption labels. Companion to the desk research in #669. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prose only. Every label, citation, table row and verdict is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
omesser
force-pushed
the
spike/668-harness-tools-under-acp
branch
from
September 16, 2026 23:23
e93e5d9 to
ad055e2
Compare
Provider-free /tools for hermes, an xAI model override for opencode, and the two named rows the matrix never had. Corrects the elicitation.form inference: it re-renders a question, it does not switch a tool on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `pi v0.85.1` two earlier probes recorded is a session banner, not an answer. With the local model server running pi lists read, bash, edit, write, which is its whole documented set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
omesser
commented
Sep 17, 2026
omesser
left a comment
Owner
Author
There was a problem hiding this comment.
CLEARED @ 6325d56 — docs-only measured companion to #669 / #668.
Spec (#668 Done-when)
- Research note with Fact/Inference/Assumption table ✓
- Per-Harness matrix (7 rows, measured where possible) ✓
- Ranked options with ADR consequences ✓
- Smallest falsifying probe for top option (cwd + Pro/Max computer-use) ✓
- Follow-ups already filed for surviving options (#782–#786); listing them on #668 next
Standards
- Docs only (
docs/research/harness-tools-under-acp-probe.md+ “since shipped” preamble on the #669 note). No product code. - Honest corrections (elicitation.form does not add a tool; turn budget 120s; self-report vs
initcaveats). - CI green; MERGEABLE/CLEAN.
Soft: tip may need rebase before land if main moved. Ready to merge when you say yes.
— Cursor agent (Architect), on @omesser's behalf.
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#668 asks whether a Harness attached over ACP keeps its tools or is reduced to chat.
docs/research/harness-native-tools-under-acp.md(#669) answered from adapter source and vendor docs and left five items open because it ran nothing against a live Harness. Two of those claims matter most: which tools the Agent SDK session lists, and whethercwddecides which of the user's MCP servers load. Both are cheap to measure, and #669 measured neither.What changed
Two files.
docs/research/harness-tools-under-acp-probe.mdis new, anchored at39fd012c. It records live probes againstclaude-agent-acp0.78.0 (Agent SDK 0.3.270),grok agent stdio,hermes acp,opencode acpandpi-acp, with every claim labelled Fact, Inference, or Assumption. It carries the eight answers, the per-Harness matrix, the ranked options with their ADR-0003, ADR-0018 and ADR-0023 consequences, and the smallest probe left to run.docs/research/harness-native-tools-under-acp.mdgains one "since shipped" preamble paragraph and nothing else. Its claim 14 cites a 20-second turn budget, which wasmodel::TIMEOUTat its anchore070216. #695 gave a Harness turnharness::TURN_TIMEOUTof 120 seconds (src-tauri/src/harness.rs:71at39fd012c, for #690). Perdocs/agents/docs.mdan anchored note keeps its record and gets a preamble line when a finding has since shipped, so the 20s prose below it stays as written.Measured, off the Agent SDK
initmessage through the adapter'semitRawSDKMessagesextension:claudeattach keeps 33 built-in tools,WebSearchandWebFetchincluded.AskUserQuestionandcomputer-useare the only absences.cwd. A local-scope server loaded only whencwdwas the directory it was added in. That is thecwdclaim docs(harness): Research which Harness tools survive an ACP attach #669 inferred, now observed.computer-useserver appears, with or without the undocumentedALLOW_ANT_COMPUTER_USE_MCP=1. Anthropic's page names four gates, and this machine's Team plan trips one before the interactive gate can be observed, so that cell stays an Assumption with a probe filed for it.Round two (2026-09-17) closed the rows round one could not reach, all on this Mac.
hermes: 27 tools. Measured through its own
/toolsslash command over ACP, which needs no model call, so the dead local provider stopped mattering.opencode: 10 tools (
bash, edit, glob, grep, read, skill, task, todowrite, webfetch, write). Reached by overriding the configured model toxai/grok-4.6in a copy of the config; this machine hasXAI_API_KEY.webfetchyes, no web-search tool, which matches the vendor's "conditional".cursor-agent: 19 tools and codex: 22 tools. Both are named rows in
harness.rsthat round one's matrix did not cover at all. codex is the only Harness besides grok listing an ask-the-user tool.pi: 4 tools (
read,bash,edit,write). Its local model server was the thing that was down; once it was up, pi answered. Thepi v0.85.1the earlier note recorded is a session banner printed before the turn, not pi's answer, so the tool list is on the last line ofAGENT_TEXTrather than the first. Measured twice with different prompts, identical both times, andpi --helpopens "pi - AI coding assistant with read, bash, edit, write tools". Claim 17 goes Assumption to Fact: pi keeps its whole documented set under attach. Separately,pi-acpadvertisesmcpCapabilities: {http: false, sse: false}.That also gives a rule worth keeping: a probe that returns only a banner is evidence about the provider, not about the Harness.
Claim 7 is now a Fact. A fixture directory holding only a
.mcp.jsonputprobe-project-scopeintoinit.mcp_servers. Project scope keys offcwd, measured rather than inferred.One correction, and it re-ranks an option. Round one inferred that advertising
elicitation.formswitches on exactly one tool. Measured withclientCapabilities: {"elicitation":{"form":true}},init.toolsis the same 33 names as with{}andAskUserQuestionis in neither. The adapter's own comment says why: AskUserQuestion arrives as a normal permission check, and the capability re-renders it as a form. It recovers no tool. Ranked option 2 said "it is the one client capability that switches a tool on" and now says what it actually buys, which leaves option 1 as the only tool-recovery option in the list.Also refined: hermes lists all ten
browser_*tools while the start-up CDP check marks them unavailable for the turn. A listed tool is not a callable tool on that Harness.Every measurement agrees with the merged note's Facts. The note also records two corrections to what the repository says elsewhere, neither edited here. ADR-0003's Consequences name one of four computer-use gates, and
README.md:159's "Chat-only (no MCP)" reads as a claim about pi's own tools.By design this PR has no product code, no README or ADR edit, and no
_meta.claudeCode.optionshatch in the product. The probe client is embedded in the note so a reviewer can rerun every measurement without ai-buddy code.Follow-up issues filed on V1 for the options that survived: #782 (attach
cwd), #783 (elicitation.form), #784 (README tool classes), #785 (ADR-0003 consequences), #786 (computer-use probe on a Pro or Max Mac).How to verify
Rerun the two Claude probes from the note's last section with two
cwdvalues and diffINIT mcp_servers. On this machine the second run addedcaveman-shrinkand nothing else changed.Round two's three techniques are written up in the note's new "Round two" section with runnable commands:
/toolsfor hermes, anOPENCODE_CONFIGmodel override for opencode, and a.mcp.jsonfixture directory for claim 7. Theelicitation.formprobe is a one-line change to the client, also shown.pre-commit run --files docs/research/*.mdpasses.node --test tests/*.test.js238 pass, 0 fail.Notes
Does not close #668. The owner accepts the spike.
— Claude Code agent, on @omesser's behalf.
🤖 Generated with Claude Code